{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 问题描述\n",
    "\n",
    "数据：Million Song Dataset(MSD)\n",
    "\n",
    "https://labrosa.ee.columbia.edu/millionsong/\n",
    "    \n",
    "作业使用的数据集是公开音乐数据集 Million Song Dataset(MSD) ， 它 包 含 来 自 SecondHandSongs dataset 、 musiXmatch dataset、Last.fm dataset、Taste Profile subset、 thisismyjam-to-MSD mapping、tagtraum genre annotations 和 Top MAGD dataset 七个知名音乐社区的数据。\n",
    "\n",
    "原始数据集包括：\n",
    "\n",
    "1. train_triplets.txt：三元组数据（用户、歌曲、播放次数）\n",
    "\n",
    "2. track_metadata.db：每个歌曲的元数据 \n",
    "\n",
    "由于原始数据太大，作业用的数据集只是其中的子集（播放次数最多的10万个用户、播放次数最多的3万首歌曲。 \n",
    "数据预处理过程请见DataProcessing.ipynb文件，最后得到的数据文件为：triplet_dataset_sub_song_merged.csv（1千万条记录） "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 解题提示\n",
    "\n",
    "注意：\n",
    "\n",
    "1. 如果内存有限，对每个用户，可考虑只推荐最流行的5000首歌曲。从triplet_dataset_sub_song_merged.csv中抽取播放次数排在前5000的歌曲。Recommendation_Item_CF.ipynb文件中已有抽取数据的代码。\n",
    "\n",
    "2. 由于这个数据集中并没有用户对物品的显式打分，需要将播放次数转换为分数。 \n",
    "\n",
    "3. 在协同过滤中计算用户之间的相似度或物品之间的相似度时，一种方式用播放次数/分数作为用户/物品的特征表示，同课件。可考虑将播放次数变换到[0,10]区间。 \n",
    "另一种可选的表示是只要用户播放过歌曲就表示为1，否则为0（二值化），这样物品之间的相似度为播放两个歌曲的用户交集除以播放两个歌曲的用户并集： \n",
    "。 类似的，两个用户之间的相似度可用两个用户播放歌曲的交集除以两个用户播放歌曲的并集表示。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 批改标准\n",
    "\n",
    "作业要求：将triplet_dataset_sub.csv中的数据用train_test_split分成60%数据做训练，剩下40%数据做测试。\n",
    "\n",
    "1. 实现基于用户的协同过滤； （20分）\n",
    "\n",
    "2. 实现基于物品的协同过滤； （20分）\n",
    "\n",
    "3. 实现基于模型（矩阵分解/LFM）的协同过滤。（30分）\n",
    "\n",
    "4. 对每种推荐算法的推荐结果，用Top20个推荐歌曲的准确率和召回率评价推荐系统的性能。（30分）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>song</th>\n",
       "      <th>play_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4e11f45d732f4861772b2906f81a7d384552ad12</td>\n",
       "      <td>SOCKSGZ12A58A7CA4B</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4e11f45d732f4861772b2906f81a7d384552ad12</td>\n",
       "      <td>SOCVTLJ12A6310F0FD</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4e11f45d732f4861772b2906f81a7d384552ad12</td>\n",
       "      <td>SODLLYS12A8C13A96B</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4e11f45d732f4861772b2906f81a7d384552ad12</td>\n",
       "      <td>SOEGIYH12A6D4FC0E3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4e11f45d732f4861772b2906f81a7d384552ad12</td>\n",
       "      <td>SOFRQTD12A81C233C0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       user                song  play_count\n",
       "0  4e11f45d732f4861772b2906f81a7d384552ad12  SOCKSGZ12A58A7CA4B           1\n",
       "1  4e11f45d732f4861772b2906f81a7d384552ad12  SOCVTLJ12A6310F0FD           1\n",
       "2  4e11f45d732f4861772b2906f81a7d384552ad12  SODLLYS12A8C13A96B           3\n",
       "3  4e11f45d732f4861772b2906f81a7d384552ad12  SOEGIYH12A6D4FC0E3           1\n",
       "4  4e11f45d732f4861772b2906f81a7d384552ad12  SOFRQTD12A81C233C0           2"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 读入数据\n",
    "df_data = pd.read_csv('triplet_dataset_sub.csv', encoding='latin-1')\n",
    "\n",
    "df_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 将 user 和 song 的字符串转化成数字"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "le = preprocessing.LabelEncoder()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# User 数字化后的小字典\n",
    "le.fit(df_data['user'].values)\n",
    "user = le.transform(df_data['user'].values) \n",
    "df_data_user = pd.DataFrame(data=user, columns=['user'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Song 数字化后的小字典\n",
    "le.fit(df_data['song'].values)\n",
    "song = le.transform(df_data['song'].values) \n",
    "df_data_song = pd.DataFrame(data=song, columns=['song'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将 play_count 映射到区间 [0,5]\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "mm = MinMaxScaler(feature_range=(0,5))\n",
    "col_label = ['play_count']\n",
    "mm_df_data = mm.fit_transform(df_data[col_label])\n",
    "\n",
    "# play_count 的小字典\n",
    "df_data_count = pd.DataFrame(data=mm_df_data, columns=['rating'], index =df_data.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>song</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>246</td>\n",
       "      <td>4</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>246</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>246</td>\n",
       "      <td>10</td>\n",
       "      <td>0.002832</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>246</td>\n",
       "      <td>12</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>246</td>\n",
       "      <td>23</td>\n",
       "      <td>0.001416</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user  song    rating\n",
       "0   246     4  0.000000\n",
       "1   246     5  0.000000\n",
       "2   246    10  0.002832\n",
       "3   246    12  0.000000\n",
       "4   246    23  0.001416"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Join dics\n",
    "df_data = pd.concat([df_data_user, df_data_song, df_data_count], axis = 1, ignore_index=False)\n",
    "df_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 37519 entries, 0 to 37518\n",
      "Data columns (total 3 columns):\n",
      "user      37519 non-null int64\n",
      "song      37519 non-null int64\n",
      "rating    37519 non-null int64\n",
      "dtypes: int64(3)\n",
      "memory usage: 879.4 KB\n"
     ]
    }
   ],
   "source": [
    "df_data_int = df_data.astype('int64')\n",
    "df_data = df_data_int\n",
    "df_data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>song</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>246</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>246</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>246</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>246</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>246</td>\n",
       "      <td>23</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user  song  rating\n",
       "0   246     4       0\n",
       "1   246     5       0\n",
       "2   246    10       0\n",
       "3   246    12       0\n",
       "4   246    23       0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 测试集训练集分离"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train, test = train_test_split(df_data, test_size = 0.40, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = train.values\n",
    "test_data = test.values\n",
    "data = df_data.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 基于用户的协同过滤"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# User based CF:\n",
    "from RS_User_CF_new import UserBasedCF\n",
    "\n",
    "#声明一个UserBased推荐的对象  \n",
    "User = UserBasedCF(train_data,test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "168 \t 0.0\n",
      "588 \t 0.0\n",
      "714 \t 0.0\n",
      "204 \t 0.0\n",
      "470 \t 0.0\n",
      "416 \t 0.0\n",
      "523 \t 0.0\n",
      "259 \t 0.0\n",
      "141 \t 0.0\n",
      "136 \t 0.0\n",
      "3 \t 0.0\n",
      "251 \t 0.0\n",
      "59 \t 0.0\n",
      "66 \t 0.0\n",
      "61 \t 0.0\n",
      "348 \t 0.0\n",
      "228 \t 0.0\n",
      "2 \t 0.0\n",
      "786 \t 0.0\n",
      "436 \t 0.0\n"
     ]
    }
   ],
   "source": [
    "# 以测试集第5个位置的用户为例，给出Top20歌曲的推荐列表\n",
    "User.UserSimilarity()\n",
    "recommedDic = User.Recommend(test['user'].values[4],K=3,N=20)\n",
    "for k,v in recommedDic.items():  \n",
    "    print (k,\"\\t\",v  )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              recall           precision\n",
      "             11.326%             10.885%\n"
     ]
    }
   ],
   "source": [
    "# 在测试集上给出Top20个推荐歌曲的准确率和召回率（由于作业要求没有规定相关用户的个数K，这里选K=3,也可以去其他的值进行评估）\n",
    "print(\"%20s%20s\" % ('recall','precision'))\n",
    "recall,precision = User.RecallAndPrecision(train=None,test=None,K=3,N=20)\n",
    "print(\"%19.3f%%%19.3f%%\" % (recall * 100,precision * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 基于物品的协同过滤"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Item based CF:\n",
    "from RS_Item_CF_new import ItemBasedCF\n",
    "\n",
    "#声明一个ItemBased推荐的对象 \n",
    "Item = ItemBasedCF(train_data,test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "212 \t 0.0\n",
      "149 \t 0.0\n",
      "766 \t 0.0\n",
      "70 \t 0.0\n",
      "3 \t 0.0\n",
      "97 \t 0.0\n",
      "215 \t 0.0\n",
      "52 \t 0.0\n",
      "157 \t 0.0\n",
      "76 \t 0.0\n",
      "2 \t 0.0\n",
      "168 \t 0.0\n",
      "255 \t 0.0\n",
      "449 \t 0.0\n",
      "516 \t 0.0\n",
      "702 \t 0.0\n",
      "114 \t 0.0\n",
      "398 \t 0.0\n",
      "416 \t 0.0\n",
      "756 \t 0.0\n"
     ]
    }
   ],
   "source": [
    "# 以测试集第5个位置的用户为例，给出Top20歌曲的推荐列表\n",
    "Item.ItemSimilarity()\n",
    "recommedDic = Item.Recommend(test['user'].values[4],K=3,N=20)\n",
    "for k,v in recommedDic.items():  \n",
    "    print (k,\"\\t\",v  )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              recall           precision\n",
      "             12.653%             12.160%\n"
     ]
    }
   ],
   "source": [
    "# 在测试集上给出Top20个推荐歌曲的准确率和召回率\n",
    "print(\"%20s%20s\" % (\"recall\",'precision'))\n",
    "recall,precision = Item.RecallAndPrecision(train=None,test=None,K=3,N=20)\n",
    "print(\"%19.3f%%%19.3f%%\" % (recall * 100,precision * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**结果分析：**\n",
    "\n",
    "准确率和召回率的结果反映，基于物品的协同过滤(ItemCF)优于基于用户的协同过滤（UserCF）。\n",
    "\n",
    "由于此项目是对用户进行音乐的推荐，一般来讲用户的兴趣是比较固定和持久的，而且音乐的更新速度往往不会特别快。\n",
    "在这种情况下，利用用户历史行为作推荐，比较合理。并且结合 ItemCF 算法 ‘对新用户友好，对新物品不友好’ 的特点，也是应该使用基于用户的协同过滤更加合理。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
